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required  for  controllers  to  notice  that  the  advisory  is  present,  to 
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EXECUTIVE  SUMMARY 


Conflict  Resolution  Advisory  (CRA)  is  an  automated  software  aid  for 
air  traffic  control  specialists  at  air  route  traffic  control 
centers  (ARTCCs) .  CRA  calculates,  validates,  and  displays  to  the 
en  route  controller  a  single  resolution  for  predicted  separation 
violations  detected  by  the  conflict  alert  (CA)  function.  This 
simulation  study  was  conducted  to  determine  controller  response 
time  to  a  CRA  message.  The  response  time  is  the  total  time 
required  for  controllers  to  notice  that  the  advisory  is  present,  to 
read  and  comprehend  the  text  message,  and  to  decide  that  the 
resolution  is  acceptable. 

In  Week  1  of  the  study,  three  full  performance  level  air  traffic 
controllers  were  presented  with  six  different  two-hour  traffic 
scenarios  ranging  from  moderately  high  to  very  high  levels  of 
traffic  load  and  complexity.  They  were  instructed  to  use  the  CRA 
message  at  their  discretion.  The  time  between  the  onset  of  the  CRA 
message  and  the  beginning  of  the  controller's  speech  was  measured 
for  the  conflicts  in  which  the  controllers  used  CRA.  This 
procedure  yielded  too  few  data  points  to  adequately  assess 
controller  response  time. 

In  Week  2  of  the  study,  three  FPL  controllers  (who  did  not 
participate  in  Week  1)  were  presented  with  the  same  scenarios  and 
told  to  read  and  evaluate  the  CRA  messages  as  they  appeared. 
Response  time,  defined  as  the  lag  between  the  onset  of  the  CRA 
message  and  the  beginning  of  the  controller's  speech  indicating 
that  the  resolution  was  acceptable,  was  measured.  This  response 
time  varied  from  4  to  30.5  seconds  with  a  mean  response  time  of 
12.7  (s  =  6.2). 

The  overall  error  rate  was  low.  There  were  only  24  instances  (out 
of  358  occurrences  of  conflict  alert)  in  which  the  controller  read 
information  into  the  voice  tape  that  was  different  from  the  CRA 
message  that  appeared  on  the  Data  Analysis  and  Reduction  Tool 
(DART)  printout. 

Since  only  the  prototype  software  (CRAU)  was  available  for  this 
test,  the  only  other  issue  that  was  formally  addressed  was 
controller  comments  on  the  CRA  message  format.  When  asked,  half  of 
the  controllers  said  that  the  CRA  messages  were  presented  in  a 
format  that  was  easily  understood  and  that  the  CRA  message  should 
be  presented  on  the  tabular  list. 

To  calculate  the  delay  that  is  to  be  expected  between  CRA  onset  and 
pilot  response,  the  results  of  this  study  must  be  applied  to  the 
results  of  a  previous  study  (DOT/FAA/RD-91/20)  that  examined  the 
time  required  for  message  transmission  and  pilot  response.  The 
analysis  of  the  combined  data  from  these  two  studies  suggests  that 
40  seconds  should  be  considered  as  the  upper  limit  on  the  time 
expected  to  elapse  between  CRA  onset  and  the  pilot's  initial  input 
into  the  aircraft's  controls. 
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1 .  INTRODUCTION 


Conflict  Resolution  Advisory  (CRA)  is  an  automated  software  aid  for 
air  traffic  control  specialists  at  air  route  traffic  control 
centers  (ARTCCs) .  CRA  calculates,  validates,  and  displays  to  the 
en  route  controller  a  single  resolution  for  predicted  separation 
lations  detected  by  the  conflict  alert  (CA)  function.  As  with 
an*  time-critical  warning  system,  the  algorithm  must  take  into 
account  the  time  required  for  the  operator  to  use  the  system.  The 
lag  between  the  time  that  the  CRA  message  appears  on  the 
controller's  display  and  the  time  that  the  aircraft  begins  to 
maneuver  will  affect  the  number  and  type  of  potential  resolutions. 

The  purpose  of  this  experiment  was  to  determine  controller  response 
time  to  a  CRA  message.  The  response  time  is  the  total  time 
required  for  controllers  to  notice  that  the  advisory  is  present, 
read  and  comprehend  the  text  message,  and  to  decide  that  the 
resolution  is  acceptable.  This  decision  process  is  important. 
Some  resolutions  that  CRA  provides  are  not  useful  as  the  system 
cannot  take  into  account  all  of  the  information  available  to  the 
controller.  CRA  may,  for  example,  offer  a  resolution  that  turns  an 
aircraft  into  restricted  airspace  or  toward  a  thunderstorm  cell, 
since  the  CRA  logic  contains  no  information  on  restricted  airspace 
or  weather.  This  data  on  controller  response  time  will  be  used  to 
modify  existing  CRA  parameters  to  ensure  that  the  trajectory  or 
path  of  the  maneuvered  aircraft  is  modelled  correctly  by  the  CRA 
algorithm. 

This  test  was  conducted  during  the  development  phase  of  the 
enhanced  CRA  (CRAE)  software.  Since  the  prototype  version  (CRAU) 
was  used  for  this  test,  the  only  issues  that  were  formally 
addressed  were  controller  response  time  and  controller  comments  on 
the  CRA  message  format.  Also,  response  times  were  only  examined 
for  the  first  resolution  offered;  updates  to  the  original  advisory 
were  not  examined,  as  the  display  format  for  updates  will  be 
different  in  CRAE  than  it  was  for  the  prototype. 
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2 .  METHOD 


This  simulation  study  was  conducted  over  a  two  week  period  at  the 
DYSIM  laboratory  facilities  at  the  FAA  Technical  Center  in  Atlantic 
City,  NJ.  The  procedures  for  Week  1  were  the  same  as  for  Week  2, 
with  the  exception  of  a  change  in  the  instructions  to  the  test 
controllers.  In  the  first  week,  the  controllers  were  instructed  to 
use  the  CRA  display  based  on  the  currently  anticipated  procedure; 
use  of  the  display  was  optional  and  at  the  discretion  of  the 
controller.  For  Week  2,  controllers  were  instructed  to  look  at  the 
CRA  display  when  it  appeared,  read  the  resolution  aloud  and  state 
whether  or  not  the  resolution  was  acceptable. 

CRA  Display 

The  CRA  messages  appeared  in  the  tabular  list.  Figure  2-1  shows 
examples  of  CRA  messages  as  they  appeared  to  the  test  controllers. 
A  maximum  of  one  resolution  was  presented  per  conflict  alert. 
Resolutions  can  involve  vertical  or  horizontal  maneuvers.  A 
vertical  maneuver  can  consist  of  a  climb,  a  descent,  or  a  maintain 
(i.e.,  a  level-off  maneuver  when  the  aircraft  is  already  either 
climbing  or  descending) .  The  controllers  were  able  to  change  the 
location  of  the  tabular  list  on  the  screen  display. 
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FIGURE  2-1.  EXAMPLES  OF  CRA  MESSAGES 


Example  1:  Single  horizontal  maneuver  required 

Conflict  Alert 

. AAL210  UAL202  01  041 

R2  AAL210 
R  UAL2023  R304 


Example  2:  Single  vertical  maneuver  required 

Conflict  Alert 

. AAL210  UAL202  01  04 

R  AAL210 

R  UAL202  T2905 


Example  3:  Joint  maneuver  required 

Conflict  Alert 

. AAL210  UAL202  01  04 

R  AAL210  R30 
R  UAL202  L30 


Example  4:  Multiple  pair  conflict,  one  maneuver  required 

Conflict  Alert 

. AAL210  UAL202  04  04 

. UAL202  COA301 
M6 

R  AAL210 

R  UAL202  1200 

R  COA301 


1  Conflict  Alert  tabular  list  with  intersector  notation 

2  "R"  for  single  maneuver  required  (appears  in  CRAU  only)  or 
"J"  for  Joint  (two  maneuvers  required) 

3  AID  for  CRA  aircraft 

4  Horizontal  maneuver  -  right  or  left  turn  in  degrees 

5  Vertical  maneuver  -  climb,  descend,  or  maintain 

6  CRA  designator  for  multiple  pair  conflict 
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3 .  WEEK  1 


3 . 1  METHOD 

Subjects 

Subjects  were  selected  from  a  list  of  volunteers  from  the  Ft.  Worth 
ARTCC .  They  were  three  full  performance  level  (FPL)  controllers 
who  currently  work  in  the  sectors  that  were  simulated  for  this 
study . 

Airspace 

The  airspace  simulated  consisted  of  three  contiguous  test  sectors. 
Two  of  these  sectors  were  high  altitude  sectors  (Dallas  High  and 
Ardmor  High)  and  the  third  was  a  low  altitude  sector  (Frisco  Low) . 
These  test  sectors  were  supported  by  a  controller  on  a  "ghost" 
sector.  The  ghost  sector  initiated  and  received  aircraft  but  was 
not  a  test  sector. 

Scenarios 


Six  different  scenarios,  each  approximately  1  1/2  to  2  hours  in 
duration,  were  developed  based  on  actual  traffic  from  the  sectors 
chosen  for  simulation.  To  insure  the  occurrence  of  conflicts,  the 
scenarios  were  constructed  to  range  from  moderately  high  to  very 
high  workload  (in  terms  of  number  of  aircraft  and  traffic 
complexity) .  The  scenarios  contained  emergencies  and  other  unusual 
conditions  (e.g. ,  no  radio)  and  one  of  the  scenarios  contained 
significant  weather  conditions.  Simulated  aircraft  were  operated 
by  experienced  DYSIM  pilots.  The  scenario  developers  worked  with 
the  DYSIM  pilots  and  were  able  to  make  immediate  adjustments  to  the 
traffic  loads,  as  necessary. 

Procedure 


Controllers  were  instructed  to  work  as  they  normally  would  in  that 
sector,  following  standard  operating  procedures.  Each  controller 
worked  alone  as  there  were  no  D-side  (radar  associate)  controllers. 
Before  the  first  test  session,  the  controllers  were  briefed  on  the 
functions  and  limitations  of  the  CRA  prototype  program  and  were 
given  a  description  of  the  display.  The  prototype  software 
displayed  the  resolution  on  the  tabular  list  from  the  onset  to  the 
offset  of  the  conflict  alert.  The  test  controllers'  instructions 
were  based  on  the  currently  anticipated  procedure,  that  is,  that 
use  of  the  CRA  display  is  to  be  considered  optional  and  at  the 
discretion  of  the  controller  (Floyd  Etherton,  ATP-130,  personal 
communication) . 

Two  different  scenarios  were  run  each  day  (between  6:00  p.m.  and 
midnight)  over  three  days  for  a  total  of  six  experimental  sessions. 
During  each  session,  observers  who  sat  behind  the  controllers 
recorded  the  call  signs  of  the  aircraft  involved  in  the  conflict, 
the  content  of  the  CRA  message,  and  the  controller  comments  on  the 
CRA  message.  The  observers  also  recorded  whether  the  controllers 
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used  the  CRA  display;  this  was  judged  to  be  the  case  if  the 
controllers  looked  at  the  message  and  issued  the  same  clearance  to 
the  aircraft.  The  communication  frequencies  were  recorded  on 
audiotape  so  that  the  time  between  the  onset  of  the  CRA  message  and 
the  onset  of  controller  speech  as  they  issued  the  CRA  maneuver 
could  be  measured.  The  time  of  the  onset  and  the  content  of  the 
CRA  message  was  taken  from  the  Data  Analysis  and  Reduction  Tool 
(DART)  printout. 

After  each  test  session,  controllers  completed  questionnaires  that 
asked  for  their  opinions  on  the  realism  of  the  scenarios  and  the 
perceived  workload  of  the  scenarios.  After  the  controllers' 
participation  in  the  study  was  completed,  they  filled  out  a 
questionnaire  that  asked  for  additional  written  comments  on  the  CRA 
message  format  and  the  realism  of  the  simulation. 

Only  the  instances  in  which  the  controller  used  the  CRA  message 
were  used  to  determine  the  controller  response  time  to  the  CRA 
message.  Since  the  purpose  of  the  study  was  to  determine  the 
component  of  the  delay  parameter  in  the  CRA  algorithm  attributable 
to  controller  response  time,  instances  in  which  the  controller 
would  not  use  CRA  were  not  applicable. 

3 . 2  RESULTS 

Questionnaire  Analysis 

Controllers  rated  the  realism  of  each  scenario  on  a  scale  of  one  to 
five.  A  score  of  one  was  equivalent  to  "very  unrealistic",  three 
was  "moderately  realistic"  and  five  was  "very  realistic".  The 
arithmetic  mean  (average)  of  all  six  scores  collapsed  across 
scenarios  was  3.4  (standard  deviation  (s)  =  1.4).  Subjective 
workload  measures  of  the  scenarios  consisted  of  measures  of  the 
amount  and  complexity  of  the  traffic  presented.  Controllers  rated 
the  traffic  load  and  complexity  compared  to  what  they  normally  see 
on  that  sector.  Traffic  load  estimates  ranged  from  50%  to  150% 
across  scenarios  with  a  mean  of  99%  (s  =  24).  Measures  of  traffic 
complexity  ranged  from  one  ("very  simple")  to  five  ("very  complex") 
with  a  mean  of  3.9  (s  =  1.2). 


Voice  Tape  Analysis 

In  the  16  hours  of  testing  in  Week  1,  there  were  16  instances  in 
which  the  controller  issued  the  CRA  maneuver  after  it  appeared  on 
the  display.  For  two  of  these  instances,  there  was  a  malfunction 
that  caused  a  delay  in  the  CRA  display  so  that  the  resolution 
appeared  seconds  after  the  onset  of  the  conflict  alert.  The 
remaining  14  instances  in  which  the  CRA  display  worked  properly 
were  analyzed  for  controller  response  time. 

Of  the  14  CRA  resolutions  that  the  controllers  used,  seven  involved 
descend  maneuvers,  six  were  maintain  maneuvers,  and  one  involved  a 
right  turn. 
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The  time  between  the  onset  of  the  CRA  message  and  the  onset  of 
controller  speech  varied  from  2  to  40  seconds.  The  mean  response 
time  was  18.4  seconds  (s  =  10.4)  and  the  median  (response  time  at 
the  50th  percentile)  was  16.7.  The  individual  differences  in 
response  times  were  notable.  One  controller  used  CRA  three  times 
during  the  course  of  the  study  and  the  mean  response  time  for  that 
controller  was  six  seconds.  Another  controller  used  CRA  seven 
times  with  a  mean  response  time  of  24  seconds.  The  third 
controller  used  CRA  four  times  with  a  mean  response  time  of  18 
seconds. 

Because  the  voluntary  use  of  CRA  during  Week  1  of  the  study 
resulted  in  too  few  data  points  to  adequately  determine  response 
time,  the  procedure  was  changed  for  Week  2. 
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WEEK  2 


4 . 1  METHOD 

Subjects 

Three  full  performance  level  controllers  were  selected  for 
participation  in  the  same  manner  as  described  for  Week  1 .  They 
currently  work  in  the  sectors  that  were  simulated  for  this  study 
and  had  not  participated  in  Week  1. 

Airspace  and  Scenarios 

The  airspace  simulated,  and  the  scenarios  used,  were  the  same  as 
described  for  Week  1,  with  one  exception.  More  traffic  was  added 
to  one  of  the  sectors  after  the  controller  from  Week  1  and  the 
scenario  developers  thought  that  the  sector  was  not  as  busy  as  the 
other  two  sectors. 

Procedure 


The  procedure  for  Week  2  was  identical  to  that  for  Week  1  with  one 
exception.  The  instructions  to  the  controllers  were  to  look  at  the 
CRA  display  when  it  appeared,  read  the  resolution  into  the  tape 
(i.e.,  after  keying  the  microphone),  and  state  whether  or  not  the 
resolution  was  acceptable.  The  instructions  stressed  that  the 
controller  did  not  have  to  use  the  resolution  offered  by  CRA,  even 
if  he/she  found  the  resolution  acceptable.  A  resolution  was  to  be 
judged  acceptable  as  long  as  it  was  perceived  to  be  an  effective 
solution. 

4 . 2  RESULTS 

Questionnaire  Analysis 

Controllers  rated  the  realism  of  each  scenario  on  a  scale  of  one  to 
five.  A  score  of  one  was  equivalent  to  "very  unrealistic",  three 
was  "moderately  realistic"  and  five  was  "very  realistic".  The  mean 
score  collapsed  across  each  scenario  was  3.9  (s  =  .97) .  Subjective 
workload  measures  of  the  scenarios  consisted  of  measures  of  the 
amount  and  complexity  of  the  traffic  presented.  Controllers  rated 
the  traffic  load  and  complexity  compared  to  what  they  normally  see 
on  that  sector.  Traffic  load  estimates  ranged  from  60%  to  150% 
across  scenarios  with  a  mean  of  105%  (s  =  23)  .  Measures  of  traffic 
complexity  ranged  from  one  ("very  simple")  to  five  ("very  complex") 
with  a  mean  of  3.4  (s  =  1.1). 
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Voice  Tape  Analysis 


There  were  358  occurrences  of  conflict  alert  encountered  by  the 
three  controllers  during  the  12  hoars  of  testing  in  Week  2.  Table 
4-1  categorizes  the  controllers'  responses  to  these  alerts.  CRA 
messages  generated  as  a  result  of  a  nuisance  conflict  alert  or  when 
all  of  the  conflict  aircraft  were  not  in  voice  communication  with 
the  test  controller  (i.e.,  under  track  control  only)  were  ignored 
by  the  controllers.  An  additional  60  CAs/CRAs  were  not  commented 
on  by  controllers.  In  38  instances  of  CA,  CRA  did  not  offer  a 
resolution.  This  left  125  instances  of  CRA  where  the  resolution 
was  evaluated  by  controllers  as  either  acceptable  or  not 
acceptable. 


TABLE  4-1.  CONTROLLER  RESPONSES  TO  CA/CRA 


Controller  Response 

Freauencv 

Resolution  Acceptable 

90 

Nuisance  Alert 

90 

No  Comment  on  CA/CRA 

60 

Aircraft  Under  Track  Control  Only 

45 

No  Resolution  Available 

38 

Resolution  Unacceptable 

35 

Total 

358 

Acceptable  Resolutions.  In  25%  of  the  total  CAs  (72%  of  the 
125  CRA  messages  evaluated  by  controllers) ,  the  resolution  was 
considered  acceptable.  In  5  of  these  90  instances,  there  was 
a  malfunction  that  caused  a  delay  in  the  CRA  display  so  that 
the  resolution  appeared  seconds  after  the  onset  of  the 
conflict  alert.  The  remaining  85  instances  in  which  the  CRA 
display  worked  properly  were  analyzed  for  controller  response 
time.  Of  the  85  responses  analyzed  for  response  time,  34  were 
attributable  to  one  test  controller,  29  to  another,  and  22  to 
the  third. 
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The  time  between  the  onset  of  the  CRA  message  and  the  onset  of 
controller  speech  indicating  an  acceptable  resolution  was 
measured.  For  example,  if  the  controller's  response  was,  "CRA 
wants  to  turn  United  123  right  thirty  degrees  .  .  .  yeah,  that 
will  work",  the  end  of  the  response  time  was  measured  as  the 
beginning  of  the  "yeah."  This  response  time  varied  from  4  to 
30.5  seconds  (see  Table  4-2).  The  mean  response  time  was  12.7 
seconds  (s  =  6.2).  Table  4-3  shows  the  response  times  at 
several  percentiles. 

TABLE  4-2.  CONTROLLER  RESPONSE  TIME  (IN  SECONDS) 


Minimum  4.00 
Maximim  30.50 
Mean  12.72 
Standard  Deviation  6.18 


Total  Number  of  Observations  =  85 

TABLE  4-3.  CONTROLLER  RESPONSE  TIME  (PERCENTILES) 


5th 

6.0 

10  th 

7.0 

50th 

11.0 

90  th 

22.0 

95  th 

26.0 

The  variability  among  controllers  in  terms  of  response  times 
was  small.  The  mean  response  time  for  one  controller  was  11 
seconds  (s  =  4.9),  for  another  was  13.2  (s  =  6.5),  and  for  the 
third  controller  was  14.7  (s  =  6.9). 


Nuisance  Alerts.  Twenty-five  percent  of  the  CA  instances  were 
considered  by  the  controllers  to  be  nuisance  alerts. 
Resolutions  to  nuisance  CAs  were  not  evaluated  by  the 
controllers.  Independent  from  the  controllers'  observations, 
a  CA  was  classified  as  a  nuisance  alert  by  the  controller 
observers  when  no  action  was  required  by  the  controller  to 
resolve  the  potential  conflict.  There  were  only  a  few 
instances  in  which  the  controller  classified  the  CA  as  a 
nuisance  alert  when  the  observer  did  not. 

No  Comment .  In  60  (17%)  of  the  358  instances  of  conflict 
alert  the  controller  did  not  comment  on  the  CA  or  CRA  message. 
Most  (45)  of  these  instances  were  due  to  the  controller  not 
mentioning  that  a  CA  was  occurring;  many  of  these  instances 
could  be  attributable  to  nuisance  alerts  (as  noted  by  the 
observers) .  Seven  instances  were  due  to  the  controller  noting 
that  there  was  a  CRA  message  but  giving  no  comment.  Four 
instances  were  due  to  the  CRA  message  disappearing  before  the 
controller  could  read  it  and  four  were  due  to  the  tabular  list 
being  too  cluttered  to  read  effectively. 
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Advisories  for  Aircraft  Under  Track  Control  Only.  Thirteen 
percent  of  the  CAs  involved  aircraft  that  were  not  in  voice 
communication  with  the  test  controller.  Resolutions  that  were 
presented  to  the  test  controller,  but  were  intended  for 
aircraft  under  track  control  only  (and  not  in  voice 
communication  with  the  test  controller)  were  not  evaluated  by 
the  controllers. 

No  Resolution  Available.  In  ten  percent  of  the  358  instances 
of  conflict  alert,  CRA  did  not  generate  a  resolution.  In 
these  cases,  the  spaces  that  normally  contained  the  resolution 
were  blank. 

Unacceptable  Resolutions.  In  ten  percent  of  the  total  CAs, 
the  resolution  was  considered  unacceptable.  This  does  not 
include  the  instances  in  which  the  CRA  resolution  given  to  the 
controller  involved  maneuvers  for  aircraft  not  in  voice 
communication  with  the  test  controller.  The  controllers' 
reasons  for  judging  the  resolutions  to  be  unacceptable  were 
not  examined  in  detail,  since  they  will  change  with  the 
implementation  of  CRAE. 

Controller  Response  Errors 

Occasionally,  a  controller  would  read  information  into  the  tape 
that  was  different  from  the  CRA  message  that  appeared  on  the  DART 
printout.  Since  no  video  of  the  tabular  list  was  recorded  for  this 
test,  a  discrepancy  between  what  appeared  on  the  scope  and  what  was 
on  the  DART  printout  cannot  be  ruled  out.  However,  such  a 
discrepancy  appears  improbable. 

Even  though  the  overall  error  rate  was  low,  the  errors  were 
analyzed  in  detail.  This  analysis  was  conducted  for  the  purposes 
of  trying  to  predict  what  types  of  errors  controllers  could  be 
expected  to  make  with  CRA  so  that  the  probability  of  these  errors 
could  be  minimized  in  the  future. 

There  were  24  instances  (6.7%  of  the  358  CAs)  in  which  the 
controller  read  information  that  was  different  than  what  was 
recorded  on  the  DART  printout  as  appearing  on  the  tabular  list. 
These  misreads  and  misspeaks  (i.e.,  meaning  one  thing  but  saying 
another)  are  divided  into  five  categories  and  tabulated  in  Table 
4-4. 


7This  is  only  true  for  the  prototype  software, 
message  "no  res",  meaning  no  resolution  is  available, 


In  CRAE,  the 
will  appear. 
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TABLE  4-4.  CONTROLLER  MISREADS  OF  CRA  MESSAGES 


Controller  Response  Frequency 

Misreading  the  Maneuver  9 

Misreading  the  Aircraft  Call  Sign  8 

Reporting  No  Resolution  (when  one  appeared)  4 

Assigning  the  Maneuver  to  the  Wrong  Aircraft  2 

Miscellaneous  i 

Total  24 


Misreading  the  Maneuver.  Controllers  reported  a  maneuver  that 
was  different  from  (but,  in  most  cases,  very  similar  to)  the 
CRA  message  in  37.5%  of  these  misread  instances.  A  maneuver 
was  reported  in  the  same  direction,  but  of  different  magnitude 
in  four  of  these  cases;  for  example,  "Right  40"  was  read  as 
"Right  70  degrees."  In  one  instance,  the  opposite  direction 
was  reported  ("Right  40"  was  reported  as  "Left  30").  In  two 
cases,  the  controllers  thoroughly  misread  the  CRA  message; 
"Maintain  300"  was  reported  as  "Right  on  both  aircraft"  and 
"Descend  260"  was  reported  as  "Right,  but  it  doesn't  say  how 
many  degrees."  These  last  two  cases,  in  which  the  controllers 
thought  the  maneuvers  involved  right  turns,  are  probably 
attributable  to  the  ”R”  that  indicated  that  the  resolution 
involved  only  one  maneuver  (as  opposed  to  a  joint  maneuver) . 
This  "R"  will  not  appear  in  the  CRAE  display. 

Misreading  the  Aircraft  Call  Sian.  Thirty-three  percent  of 
the  24  instances  involved  misreading  the  aircraft  call  sign. 
This  ncludes  instances  in  which  the  numbers  in  the  call  signs 
are  transposed  or  the  airline  is  misidentif  ied  with  the 
correct  flight  number.  In  none  of  these  instances  did  the 
controller  catch  or  correct  the  error. 

Reporting  No  Resolution.  Seventeen  percent  of  the  24  misread 
instances  involved  reporting  that  there  was  no  resolution  when 
there  was  one  (according  to  the  DART  printout) . 

Incorrectly  Assigning  Maneuvers.  In  two  (8%  of  the  total) 
instances,  the  controllers  assigned  the  maneuver  to  the  wrong 
aircraft.  For  example,  in  a  conflict  between  OPEC  34  and  AAL 
595,  the  controller  read  "Right  40  for  OPEC  34"  when  the 
resolution  was  "Right  40  for  AAL  595." 
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Miscellaneous.  Finally,  there  was  one  case  in  which  there 
were  two  successive  conflicts  that  involved  one  of  the  same 
aircraft.  A  few  seconds  after  there  was  a  conflict  between 
DYNAM  1  and  COA  114,  there  was  a  conflict  between  DYNAM  1  and 
COA  258.  The  controller  did  not  notice  this  change  and 
thought  that  the  new  resolution  was  an  update.  The  format  for 
presentation  of  updates  will  change  with  CRAE. 

Error  analysis  often  points  to  display  characteristics  that  induce 
them.  In  this  case,  however,  no  category  of  errors  could  easily  be 
attributed  to  any  specific  aspect  of  the  display.  The  types  and 
frequency  of  misreads  made  by  controllers  during  this  test  are  an 
inevitable  consequence  of  humans  performing  such  tasks  and  must  be 
expected  to  occur. 
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5.  POST-TEST  QUESTIONNAIRE  RESULTS 

After  the  test,  controllers  were  given  a  questionnaire  designed  to 
elicit  controller  opinion  on  the  CRA  display  and  suggestions  as  to 
how  the  simulation  could  be  improved. 


S.l  CONTROLLER  COMMENTS  ON  THE  REALISM  OF  THE  SIMULATION 

The  test  controllers  were  asked  to  rate  the  realism  of  the 
communications,  the  communications  equipment,  and  of  the  displays 
and  controls  used  for  the  test  on  a  scale  of  one  to  five.  The 
anchors  for  "one",  "three”,  and  "five"  were  "very  unrealistic", 
"moderately  realistic",  and  "very  realistic",  respectively.  The 
mean  rating  for  the  realism  of  the  communications  was  4.3.  The 
mean  rating  for  the  realism  of  the  communications  equipment  was 
3.2,  and  the  mean  rating  for  the  realism  of  the  displays  and 
controls  was  4.5. 


5.2  CONTROLLER  COMMENTS  ON  CRA 

In  response  to  the  question,  "Were  the  CRA  messages  presented  in  a 
format  that  was  easily  understood?,"  three  out  of  the  six 
controllers  said  "yes,"  two  responded  "no,"  and  one  said  that  it 
was  clear  as  long  as  there  was  only  one  conflict  present  involving 
less  than  three  aircraft. 

In  response  to  the  question,  "Should  the  CRA  message  be  displayed 
on  the  tabular  list?,"  three  of  the  six  controllers  responded 
"yes,"  one  said  "yes,  but  also  on  the  data  block,"  and  two 
controllers  responded  "no."  The  follow-up  question  asked,  "If  not, 
how  should  it  be  displayed?"  Of  the  two  controllers  who  responded 
that  the  CRA  resolution  should  not  be  displayed  on  the  tabular 
list,  one  suggested  that  the  maneuver  be  displayed  graphically  in 
a  format  similar  to  the  present  MSAW  alerts,  and  the  other 
controller  was  not  sure  how  the  resolutions  should  be  presented. 

During  the  test  and  in  the  questionnaire,  controllers  commented  on 
the  difficulties  of  having  to  look  away  from  the  conflict  to  read 
the  tabular  list.  Focussing  on  the  tabular  list  during  a  conflict 
not  only  went  against  what  they  were  accustomed  to  doing,  but  also 
seemed  attentionally  taxing,  as  they  then  had  to  switch  back  to  the 
traffic.  As  one  controller  noted,  however,  it  is  possible  that 
some  of  these  difficulties  could  be  resolved  with  increased 
experience  with  CRA,  particularly  for  new  controllers  who  have 
fewer  ingrained  habits. 
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6.  SUMMARY  AMD  CONCLUSIONS 


In  Week  1  of  the  study  it  was  found  that  the  controllers'  initial 
response  to  CRA  was  to  resolve  an  impending  conflict  and  then,  if 
there  was  time,  examine  the  CRA  message.  This  response  would  be 
expected  from  controllers  who  have  had  no  experience  with  CRA. 
Controllers  who  participated  in  Week  1  said  that  they  occasionally 
used  the  CRA  maneuver  just  to  see  what  it  would  do.  This  would 
help  to  explain  why  the  response  times  noted  for  Week  1  were  longer 
than  those  noted  for  Week  2  (although  the  paucity  of  data  points 
combined  with  the  high  variability  in  response  times  from  Week  1  do 
not  justify  such  a  comparison) . 

Week  2  showed  that  when  controllers  use  the  CRA  message,  an  average 
of  13  seconds  could  be  expected  to  be  required  for  the  controller 
to  read  the  CRA  message  and  decide  that  it  is  usable. 

A  different  use  of  CRA  from  that  noted  by  controllers  in  Week  1 
might  be  expected  for  controllers  who  have  had  previous  experience 
with  CRA.  The  confidence,  or  lack  of  confidence,  that  results  from 
experience  with  the  system  will  change  the  way  controllers  use  CRA. 
Also,  CRA  may  be  used  more  when  controllers  have  not  anticipated  a 
specific  conflict,  rather  than  when  the  conflict  alert  was  expected 
and  a  maneuver  already  planned.  Both  the  types  of  errors  noted, 
and  to  a  smaller  extent  the  response  times  noted,  would  be  expected 
to  change  (although  not  dramatically)  with  controller  experience 
with  the  CRA  function  and  display. 

The  application  of  the  results  of  this  study  must  be  considered  in 
the  context  of  the  computation  of  the  delay  parameter  to  be  used  in 
the  CRA  algorithm.  The  relevant  component  of  this  delay  concerns 
the  time  lapse  between  the  onset  of  the  CRA  message  and  the 
beginning  of  the  pilot's  response.  Considerations  of  how  much  time 
will  elapse  between  the  time  the  CRA  message  appears  and  the  time 
the  pilot  makes  an  input  into  the  controls  must  include  several 
factors.  These  factors  include  the  controller's  response  time,  the 
time  required  for  successful  transmission  of  a  controller's  message 
to  the  pilot  and  the  pilot's  response  time. 

The  appropriate  measure  of  controller  response  time  is  derived  from 
the  results  of  Week  2.  To  determine  pilot  response  time,  a  study 
was  conducted  using  voice  tapes  from  ARTCCs  (DOT/FAA/RD-91/20, 
August  1991) .  Pilot  response  time  was  combined  with  transmission 
time  and  measured  from  the  beginning  of  the  controller's 
transmission  of  a  maneuver  required  for  traffic  avoidance  to  the 
end  of  the  pilot's  correct  acknowledgement.  The  data  from  that 
study  are  summarized  in  Table  6-1  and  Table  6-2.  Personal 
communications  with  experts  (such  as  the  National  Resource 
Specialist  in  Flight  Management)  and  an  informal  pilot  study 
supported  our  assumption  that  by  the  end  of  the  verbal 
acknowledgement,  the  pilot  will  initiate  an  input  into  the  controls 
(whether  or  not  the  pilot  communicating  is  also  the  pilot  flying) . 
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TABLE  6-1.  PILOT  RE8PON8E  TINE  (IN  SECONDS) 


Minimum  4.00 
Maximum  40.00 
Mean  10.85 
Standard  Deviation  5.91 


Total  Number  of  Observations  =  80 


TABLE  6-2.  PILOT  RE8PON8E  TIME  (PERCENTILES) 


5th 

5.0 

10th 

6.0 

50th 

9.0 

90  th 

17.0 

95  th 

22.5 

It  is  possible  to  add  the  mean  or  90th  percentile  pilot  response 
time  (RT)  to  the  mean  or  90th  percentile  controller  response  time. 
However,  a  more  appropriate  statistical  analysis  considers  all 
possible  pairings  of  these  two  data  sets  and  calculates  the 
percentiles  for  the  sum  of  the  response  times  over  all  pairings. 
This  method  provides  the  best  estimates  available  for  the  combined 
percentiles,  assuming  that  the  pilot  and  controller  response  times 
are  independent.  Table  6-3  presents  these  percentiles  with  the 
upper  or  lower  confidence  limits. 


TABLE  6-3.  COMBINED  CONTROLLER/ PILOT  RESPONSE  TIMES  (IN  SECONDS) 


Percentiles 

5t*> 

10  th 

90  th 

95  th 

Point  Estimates  (in  seconds)  13 

14 

35 

40 

Confidence  Limits: 

Lower  . 05 

12 

13.5 

Lower  . 10 

12.5 

14 

— 

— 

Upper  . 10 

— 

— 

37.5 

43 

Upper  . 05 

— 

—  — 

38.5 

44 

Mean  Total  Response  Time 
95%  Confidence  Limits 

= 

23.57 

= 

(21.69, 

25.45) 

90%  Confidence  Limits 

= 

(22.00, 

25.10) 

Based  on  these  data. 

it 

is  recommer  *ed  that 

40  seconds  be 

considered  for  the  upper  limit  on  the  'me  expected  to  elapse 
between  the  onset  of  the  CRA  display  and  the  time  the  pilot  makes 
an  input  into  the  aircraft's  controls. 
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